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EXECUTIVE SUMMARY 



The federal Workforce Investment Act (WIA) of 1998 superceded the earlier Job 
Training Partnership Act and authorizes a new workforce investment system. As a result of WIA 
legislation, states and localities formed Workforce Investment Boards (WEBs) and workforce 
development regions. The workforce development services offered to in-school and out-of- 
school youth in West Virginia’s Region I are the subject of this evaluation report. 

West Virginia WIA Region I, encompasses Fayette, Greenbrier, McDowell, Mercer, 
Monroe, Nicholas, Pocahontas, Raleigh, Summers, Webster, and Wyoming counties. This region 
includes all school districts falling under the purview of West Virginia’s Regional Education 
Service Agency I (RESA I) and several school districts in RESA IV, as well. 

The Region I workforce investment board (R1WIB) approved the Region I plan in April 
2001. According to federal mandate, youth services programs funded by WIA are to assist youth 
in meeting employment or educational goals they articulate in collaboration with staff. Programs 
for youth must include tutoring, study skills training, and instruction leading to secondary school 
completion; mentoring; paid and unpaid work experience; occupational skills training; leadership 
development; summer employment opportunities; and other support services. Participants are to 
receive counseling and follow-up services as appropriate, as well. 

This end-of-year evaluation was primarily performed via the conduct of an interim staff 
interview to collect formative data, and via the administration of client (in- and out-of-school 
participants) and employer satisfaction surveys to collect summative information. Assessment of 
other federal performance objectives was not possible as their calculation is dependent on the 
number of clients exiting the program; this early in the program’s implementation, no clients 
have yet exited. 

Conclusions from this evaluation were that in- and out-of-school youth and employers 
were satisfied with the program. Youth respondents reported that RESA I staff were 
knowledgeable, helpful and respectful, and employers indicated that they would recommend the 
program to other employers and planned to continue their participation next year. On the other 
hand, youth would have preferred more tutoring in specific content areas, assistance with job 
searches, preparation for post-secondary life, more work experiences, and better compensation. 
Also, important to employability in remote rural areas, some participants requested assistance in 
obtaining a vehicle. Employers sought an easier process for program involvement and reported 
that some youth needed further training before participation in work experiences. 

RESA I staff were pleased with the program’s progress and their own collegial and 
cooperative organizational culture. They were less pleased with slow Job Service response and 
the large caseloads for which they were responsible. 
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Recommendations offered by the evaluator, and based upon the findings, include 
continuing to offer youth access to knowledgeable staff; ensuring that youth receive tutoring in 
specific subject areas as needed, assistance with job searches, and greater preparation for post- 
secondary experiences; and investigating students’ concerns about work hours and compensation. 
In terms of employers, it was suggested by the evaluator that RESA I staff offer training to youth 
in areas employers report to be of concern, including social skills, work habits, and specific skill 
sets. RESA I should also consider establishing structures and processes that make employer 
participation simpler. In addition, the evaluator suggested that RESA I staff contact other WIA 
programs to investigate strategies for addressing slow Job Service turnover. 



INTRODUCTION 



Federal Workforce Investment Act Legislation 

The federal Workforce Investment Act (WLA) of 1998 superceded the earlier Job 
Training Partnership Act and authorizes a new workforce investment system (U.S. Department of 
Labor, 1998). This system includes state and local workforce investment boards and local youth 
councils who plan and monitor implementation of services to customers. Local boards, 
moreover, designate one-stop (locations at which all core services are available) operators and 
training providers (Boshara, Scanlon, & Page-Adams, 2000; Johnson, 2000). WLA is intended to 
consolidate myriad and often loosely-coupled, if at all coordinated, federal job training and 
workforce development programs. It is also intended to encourage localities to define their own 
needs and priorities (Beaulieu, 2000). 

Via WLA support, one-stop centers provide a variety of employment, career training, and 
education services to youth and adults (Imel, 1999; Pantazis, 1999). Youth services are to assist 
youth in meeting employment or educational goals they articulate in collaboration with staff. 
Programs for youth must include tutoring, study skills training, and instruction leading to 
secondary school completion; mentoring; paid and unpaid work experience; occupational skills 
training; leadership development; summer employment opportunities; and other support services. 
Participants are to receive counseling and follow-up services as appropriate, as well. 

Ultimately, WIA is intended to restructure the entire U.S. workforce development system 
(Buck, 2002). According to Buck’s analysis of early implementation of the Act, WIA advances 
five major goals to achieve this end: 

• Streamlining services through a One-Stop service delivery system involving 
mandated public sector partners; 

• Providing universal access for all job seekers, workers and employers; 

• Promoting customer choice through use of vouchers and consumer report cards on 
the performance of training providers; 

• Strengthening accountability by implementing stricter and longer-term 
performance measures; and 

• Promoting leadership by the business sector on state and local Workforce 
Investment Boards (WIBs) (2002, p. 3). 

The above goals, in turn, provide structure to state and local implementations of WLA. 

West Virginia Workforce Development 

On August 8, 2000, West Virginia’s then-governor, Cecil Underwood, signed an 
executive order to comply with the federal WIA mandate (Governor’s Workforce Investment 
Office, 2000). As a result, a state workforce investment board was created, as were local boards 
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and youth councils. Regions were also established within which one-stop centers (WORK4WV) 
were to serve local clients. 

The Governor’s Workforce Investment Office, in conjunction with the state Workforce 
Investment Board, provides oversight and guidance to regional programs. In addition, the state 
Workforce Investment Office is charged with assisting regions’ efforts to streamline workforce 
development and other services. Ultimately, such coordination is intended to ensure that clients 
experience comprehensive, seamless service. 

In late 2001, however, the state Workforce Investment Office was restructured by 
executive order of the governor (“Expert speaks on work programs,” 2001; “Agency funding on 
hold,” 2001; “Administration hears complaints,” 2001). The state Workforce Investment Office 
was transferred from the Bureau of Employment Programs to the auspices of the state 
development office, and was thenceforth called the Governor’s Workforce Investment Division. 

According to the newly created Division web site, the purpose of the Division is “to 
coordinate workforce investment programs in West Virginia. [It] is also charged with providing 
technical assistance to Local Workforce Investment Boards in the state” (2002). Further, its 
mission is 



• To create more and better jobs for West Virginians through training and 
subsequent placement of all unemployed and underemployed individuals. 

• To identify West Virginia's current and future workforce needs. 

• To coordinate business, education, and government efforts and resources. 

• To promote economic development by providing employers with easy access to a 
highly skilled workforce (2002). 

Region I Workforce Development 

West Virginia WIA Region I encompasses Fayette, Greenbrier, McDowell, Mercer, 
Monroe, Nicholas, Pocahontas, Raleigh, Summers, Webster, and Wyoming counties. This region 
includes all school districts falling under the purview of West Virginia’s Regional Education 
Service Agency I (RESA I) and several school districts in RESA IV, as well. The Region I 
workforce investment board (R1WIB) approved its plan in April 2001, after collaborating with 
Region I local elected officials in its development for several months and after public comment 
was solicited. 

The R1WIB plan proposed to draw together a variety of services to local citizens, youth, 
and employers (West Virginia Workforce Investment Board, 2001). Services thus organized are 
to be coordinated and delivered seamlessly to clients. Among the services offered by the RESA I 
centers are those directed toward youth both in and out of school. The R1WEB and its youth 
council are responsible for coordinating local youth services and ensuring that all local programs 
providing assistance to youth are involved in some capacity at the one-stop centers. 
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Initial W0RK4WV one-stop centers have been established in Beckley (13 agencies are 
situated in this location), Lewisburg, Princeton, and Summersville. Additional satellite centers 
are located in Fayetteville, Hinton, Marlinton, Union, Pineville, and Webster Springs. 

Local education agencies for each of the 11 county school districts in Region 1 submitted 
proposals to the RESA I in the spring of 2001. Proposals described the manner in which districts 
intended to collaborate with the RESA I, including plans for coordinating youth services and 
leveraging resources in the counties to achieve the board’s goals. 

Region I Counties 

The 1 1 counties constituting Region I are located in the southeastern portion of West 
Virginia and are primarily rural. According to the 2000 United States census (2002), the counties 
cover 8,372 square miles and are home to a total of 350,263 people. 

Demographic data derived from the 2000 census for the Region I counties are presented 
below in Table 1. Pocahontas and Webster counties are the least populated, with respective 
populations of 9,131 and 9,719, whereas Raleigh and Mercer are the most populated, with 79,220 
and 62,980 residents, respectively. McDowell County has experienced the largest population 
decline, losing nearly a quarter (22.4%) of its residents between 1990 and 2000. Monroe County, 
on the other hand, showed a 17.5% increase in population in the same time period. Fayette, 
Greenbrier, Nicholas, and Pocahontas counties showed the least change in population. All eleven 
counties are predominantly White, although McDowell has a sizeable population of African 
American residents (1 1.9%) that is well above the state average of 3.2%, as does Raleigh (8.5%). 
In terms of square mileage, Greenbrier County is the largest of the 11, while Summers County is 
the smallest. Mercer County is most densely populated, with 150 persons per square mile. 
Pocahontas is the least densely populated county, with only 9.7 individuals per square mile. 

Indicators of economic well-being for the Region I counties are presented in Table 2. As 
with the demographic information, these data are drawn from the 2000 United States census 
( 2002 ). 



Monroe County appears to have the highest rate of home-ownership (84.5%), whereas 
Raleigh (76.5%) and Greenbrier (76.6%) have the lowest rates. McDowell and Webster counties 
have the lowest median household incomes of the eleven counties. McDowell residents earn a 
median income of $18,592 per year, and Webster residents earn a median annual income of 
$19,533. Raleigh County residents earn the highest median income of $27,864, which is 
somewhat higher than the state median income of $27,432. It should be noted that Raleigh 
County is the only Region I county approximating the state median income; the remaining 10 
counties fall below the state median. 
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Table 1 

2000 Census Demographic Data for Region I Counties 1 



County 


2000 

popula- 

tion 


1990-2000 

popula- 

tion 

percent 

change 


Percent 
persons 
under 18 


Percent 

White 


Percent 

African 

American 


Square 

miles 


Persons 

per 

square 

mile 


Fayette 


45,579 


-0.8 


21.7 


92.7 


5.6 


644 


71.7 


Greenbrier 


34,453 


-0.7 


21.6 


95.2 


3.0 


1,021 


33.7 


McDowell 


27,329 


-22.4 


23.2 


87.1 


11.9 


535 


51.1 


Mercer 


62,980 


-3.1 


21.1 


92.6 


5.8 


420 


150.0 


Monroe 


14,583 


17.5 


20.1 


92.7 


6.0 


473 


30.8 


Nicholas 


26,562 


-0.8 


23.3 


98.8 


0.1 


649 


40.9 


Pocahontas 


9,131 


1.4 


20.9 


98.4 


0.8 


940 


9.7 


Raleigh 


79,220 


3.1 


21.5 


89.6 


8.5 


607 


130.5 


Summers 


12,999 


-8.5 


20.5 


96.6 


2.2 


361 


36.0 


Webster 


9,719 


-9.4 


23.0 


99.2 


0.0 


556 


17.5 


Wyoming 


25,708 


-11.3 


22.4 


98.6 


0.6 


501 


51.3 


West Virginia 


1,808,344 


0.8 


22.3 


95.0 


3.2 


24,078 


75.1 



'Data extracted from the 2000 U.S. Census Bureau website. 
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Table 2 

2000 Census Indicators of Economic Well-Being for Region I Counties 2 



County 


Home-ownership 

rate 


Median household 
income 


n L.u... 

rciLcm uciun 

poverty 


Percent children 
below poverty 


Fayette 


77.2 


23,578 


21.2 


31.2 


Greenbrier 


76.6 


26,800 


16.2 


24.3 


McDowell 


80.1 


18,592 


31.4 


39.3 


Mercer 


76.8 


26,279 


19.0 


29.3 


Monroe 


84.5 


26,592 


15.9 


22.3 


Nicholas 


82.8 


25,872 


20.1 


28.0 


Pocahontas 


80.3 


24,035 


17.5 


26.4 


Raleigh 


76.5 


27,864 


17.4 


25.7 


Summers 


79.1 


21,664 


24.2 


34.1 


Webster 


79.0 


19,533 


28.5 


38.9 


Wyoming 


83.3 


23,994 


23.0 


30.3 


West Virginia 


75.2 


27,432 


16.8 


24.7 



The percent of persons living below the federal poverty line and the percent of children 
living below the poverty line are useful indicators of economic conditions. With nearly a third 
(31.4%) of residents living below the poverty line, McDowell County appears to be the least 
economically healthy county in Region I. Webster County, too, has a high percentage of residents 
in poverty (28.5%). Better off are Greenbrier, Raleigh, and Pocahontas counties, with respective 
percentages of persons below the poverty line of 16.2%, 17.4%, and 17.5%. 

The percent of children living in poverty tends to be an even more dramatic indicator of 
economic health. In McDowell County, more than a third (39.3%) of youth live below the federal 
poverty line. More than a third (38.9%) of children in Webster County also live in poverty. 
Monroe County children seem to fare best of all, with 22.3% living below the federal poverty 
line. However, even this percentage, nearly a quarter, indicates that a significant portion of the 
population of children are living under strained financial circumstances. 

Overall, it appears that Region I county residents are primarily White and tend to live in 
relatively sparsely populated rural areas. A significant portion of Region I residents struggle with 



2 Data extracted from the 2000 U.S. Census Bureau website. 




14 



6 



poverty, including a large percent of children. For the most part, Region I county residents’ 
median annual income falls below that of the state median. Thus, the state context, and national 
trends as described by RESA I (2001a, 2001b), render Region I counties in need of the services 
outlined by WLA legislation. 

Evaluation of RESA I Youth Development Services 

AEL contracted with RESA I to conduct formative and summative evaluation of 
RlWIB’s youth services program from July 2001 until June 2002. AEL is a private, nonprofit 
corporation headquartered in Charleston, West Virginia. AEL has served for over 35 years as the 
regional educational laboratory for Kentucky, Tennessee, Virginia, and West Virginia, and also 
houses the Regional Eisenhower Math/Science Consortium and the Region IV Comprehensive 
Center. In addition, educators and others can receive services and information from the ERIC 
Clearinghouse on Rural Education and Small Schools at AEL. AEL also provides research, 
development, dissemination, and evaluation services on a contractual basis to national, regional, 
and local education agencies. 

The Planning, Research, and Evaluation (PRE) unit of AEL has a strong record of 
successfully completed research and evaluation studies of educational processes, products, and 
projects. PRE staff are skilled in a wide variety of quantitative and qualitative methodologies, 
including focus group interviews, survey design and analysis, and participant observation. The 
unit also possesses technology to support data collection, analysis, and presentation, from 
qualitative data management software (e.g., NUD*IST) to statistical programs (e.g., SPSS). PRE 
staff adhere to The Program Evaluation Standards in the conduct of all their work. This 
evaluation of the RESA I youth services program was conducted by one Research and Evaluation 
Specialist and one Research Assistant. AEL also provided clerical and other support services 
toward the completion of the evaluation. 

As per agreements between AEL and RESA I staff, the evaluation is to consist of both 
process and outcome measures. Process evaluation is to document the degree to which and ways 
in which the project is proceeding, and will provide staff with information for decision making. 
Hence, process evaluation is primarily formative. Data from staff interviews and document 
review will be collected for this purpose. Such formative and process evaluation will occur 
during the program’s first year of implementation. 

Outcome evaluation, primarily summative in nature, is to offer an assessment of the 
extent to which program objectives were achieved, particularly in regard to client and employer 
satisfaction. The achievement of performance goals will be calculated as stipulated by the U.S. 
Department of Labor and the West Virginia WIB. Data will be drawn from Individual Service 
Strategy (ISS) forms, RESA I's case management and data collection mechanism; from customer 
and employer satisfaction protocols; and from staff interviews. Outcome evaluation of the 
achievement of performance objectives other than client and employer satisfaction will begin in 
the program’s second year of operation, as performance objectives are based on the number of 
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clients exiting the program and none have done so in the program’s first year. However, 
evaluation of customer and employer satisfaction will be conducted in both the first and second 
years of the program. 

Quantitative and qualitative methods of data collection and analysis are to be employed in 
order to produce a more accurate and comprehensive account of the project. Surveys, interviews, 
and enumeration of the achievement of performance objectives will be conducted, for instance. 

In terms of reporting, AEL staff are to provide short, timely, user-friendly summaries, 
presentations, or oral communications of evaluation results to project staff as appropriate and 
requested. AEL contracted to produce an interim report in January 2002 of activities and 
accomplishments to date. This report was delivered to RESA I staff at the beginning of January, 
as stipulated. 

AEL was also contracted to produce a final report summarizing formative and summative 
evaluation findings for use by program staff in June 2002. This report will serve the afore- 
mentioned purpose and will include data concerning the customer and employer satisfaction 
performance objectives for both the in-school and out-of-school youth programs. 

Performance Goals 

The Region I WEB stipulated required performance goals for the service area, which are 
presented below. To expedite the evaluation and to provide RESA I staff with more quantifiable, 
rigorous, and replicable data, evaluators operationalized the performance goals. Operational 
explanations of the performance goals, drawn in part from suggestions offered by Callahan 
Consultants (2001), follow below in italics. 

Performance goals for in-school youth, ages 14-21, are as follows: 

• Skill attainment rate: At least 73% of those in-school youth served by WIA have attained 
basic skills, work readiness skills, or occupational skills while receiving services. The 
rate of skill attainment is the number of basic skill goals attained + the number of work 
readiness skills goals attained + the number of occupational skills goals attained divided 
by the number of basic skills goals set + the number of work readiness skills goals set + 
the number of occupational skills goals set by participants. 

• Diploma/GED attainment rate: Of those in-school youth who enter WIA services without 
a diploma or equivalent, at least 56% will receive one by the time they leave services. 

This does not apply to in-school youth who leave services and remain in school. The 
diploma/GED attainment rate is the number of youth who attained a secondary school 
diploma or GED by the end of the first quarter after exit, divided by the number of youth 
who exit during the quarter (except those still in secondary school at exit). 

• Placement attainment rate: At least 55% of those in-school youth served by WIA are in 
one of the following activities six months after they leave services: post-secondary 
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education, advanced training, employment, military service, or a qualified apprenticeship 
program. The placement attainment rate is the number of youth in post-secondary 
education + the number in advanced training + the number in employment + the number 
in military service + the number in qualified apprenticeships in the third quarter 
following exi t, divided by the number of youth who exit during the quarter (except those 
still in secondary schools at the time of exit). 

• Participant customer satisfaction: At least 69% of the in-school youth served by WIA will 
rate their satisfaction with the programs as above average or excellent. The participant 
customer satisfaction measure is the weighted average of participant ratings on each of 
the three federally required questions assessing overall satisfaction as reported on a 0- to 
100- point scale. Additionally, to address the literal meaning of the RES A I satisfaction 
goal, customer satisfaction will also be measured as the percentage of respondents who 
give an average rating of 5.6 (where 5.5 represents a satisfaction assessment of 
“average” ) or higher on the 10-point scale across the three federally mandated 
satisfaction questions. 

• Literacy: Qualified instructors will provide WIA participants tested at two years or more 
behind grade level with educational enhancement. Of those students, 55% will raise their 
reading level by at least one grade level before exiting services. The literacy measure is 
the number of students tested as two or more years below grade level whose reading 
levels are at least one grade level higher just prior to exit, divided by the number of 
students tested as two or more years below grade level. 

Performance goals for out-of-school youth, ages 19-21, are as follows: 

• Entered employment: At least 59% of youth served by WIA who did not have jobs prior 
to services will have a job by the time services have ended. The entered employment rate 
is the number of youth who have entered employment by the end of the first quarter after 
exit divided by the number of youth who exit during the quarter (excluding youth only in 
post-secondary education or advanced training). 

• Six-months employment retention: Of those students who are employed after exit and 
who are not enrolled in post-secondary education or advanced training, at least 78% still 
have a job six months later. The six-months employment retention rate is defined as the 
number of youth who are employed in the third quarter after exit, divided by the number 
of youth who exit during the quarter (excludes post-secondary education and advanced 
training participants). 

• Six-months earnings gain: Those youth who are employed and who are not enrolled in 
post-secondary education or advanced training will earn at least $2,255 more six months 
after exiting services than they earned during the six months prior to entering services. 

The six-months earnings gain is an average measured as average pre-program earnings 
(earnings in quarter two and quarter three prior to registration ) subtracted from average 
post-program earnings (earnings in quarter two and quarter three after exit). 

• Credential attainment: Of those youth served by WIA funds who were in employment, 
post-secondary education, or advanced training when services ended, at least 51% will 
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receive a credential within six months of exiting the services. The credential attainment 
rate is the number of youth who are in employment, post-secondary education, or 
advanced training in the first quarter after exit and who received a credential by the end 
of the third quarter after exit, divided by the number of youth who exit during the quarter. 

- Literacy: A qualified instructor will provide WIA youth tested to be reading below an 
eighth grade level. Of those youth, at least 55% will raise their reading level by at least 
one grade level before exiting the services. The literacy measure is the number of 
students tested as two or more years below grade level whose reading levels are at least 
one grade level higher just prior to exit, divided by the number of students tested as two 
or more years below grade level. 

• Participant customer satisfaction: At least 69% of the youth served by WIA will rate their 
satisfaction with the programs as above average or excellent. The participant customer 
satisfaction measure is the weighted average of participant ratings on each of the three 
federally required questions assessing overall satisfaction as reported on a 0- to 100- 
point scale. Additionally, to address the literal meaning of the RESA I satisfaction goal, 
customer satisfaction will also be measured as the percentage of respondents who give 
an average rating of 5.6 (where 5.5 represents a satisfaction assessment of "average”) or 
higher on the 10-point scale across the three federally mandated satisfaction questions. 

• Employer customer satisfaction: At least 67% of the employers who hire Region I youth 
through the WORK4WV Center will rate their satisfaction with youth as above average 
or excellent. Employer satisfaction is the weighted average of employer ratings on each 
of the three federally required satisfaction questions regarding overall satisfaction as 
reported on a 0- to 100-point scale. Additionally, to address the literal meaning of the 
RESA I satisfaction goal, employer customer satisfaction will also be measured as the 
percentage of respondents who give an average rating of 5.6 (where 5.5 represents a 
satisfaction assessment of "average” ) or higher on the 10-point scale across the three 
federally mandated satisfaction questions. 

Customer satisfaction measures will be assessed in this first year of the program’s 
operation. Both in-school youth satisfaction will be evaluated, as will out-of-school youth and 
employer satisfaction. The remaining performance objectives will not be evaluated at this time, 
however. Performance objectives are based upon client exits from the program; as no clients 
have yet exited at this early point in the program, calculation of achievement of the objectives is 
not possible. 

Purpose and Audience for This Report 

This report is intended to describe RESA I services to in-school and out-of-school youth 
provided thus far as mandated by the WIA. In addition, evaluation activities to date are 
recounted; the period for reporting ranges from June 2001 to May 2002. Primarily, however, this 
evaluation report summarizes summative data, with particular emphasis on the degree to which 
customers and employers reported they were satisfied with the program. 
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The primary audience for this report includes RESA I staff, particularly those responsible 
for leading, coordinating, and implementing youth services. The information in this report is 
intended to support their efforts toward continuous improvement and to provide them with a 
quantifiable, replicable, and rigorous account of their accomplishments thus far. Another 
important audience for this report is the R1WIB, who may find the data herein to be indicators of 
RESA I’s performance in its first year of operating WIA-funded workforce investment youth 
services. Secondary audiences may also include other workforce investment boards, one-stop 
center staff, legislators, or additional organizations committed to providing workforce 
development services to youth. 
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METHODS 

The methods employed for this evaluation of the RESA I Workforce Development Youth 
Services program are multiple and include interim staff interviews, client mail and phone 
surveys, an employer phone survey, and document review. Triangulation of data provides a more 
comprehensive description of the objects of study than might be rendered by use of a single data 
collection method. Using several data sources in order to corroborate theses is what Brewer and 
Hunter (1989) call “multi method research.” This approach posits that the strengths of each 
method will compensate for the weaknesses in others, ultimately providing a more complete 
account of that being studied. 

Interim Staff Interviews 

To ascertain staff perceptions of effective strategies, structures or processes that support 
program implementation, interim outcomes, and issues and concerns, individual interviews were 
conducted with RESA I youth services staff (see Appendix A for interview protocols). 
Recommendations for improvements were also solicited from staff. The purpose of such 
interviews was to collect formative information with which project staff might make decisions 
about their program. 

The interviews were conducted in person by the evaluator on November 20, 2001 at the 
Beckley WORK4WV center with six staff members. A seventh interview was conducted by 
telephone (and tape recorded) on the same day. The interview sample consisted of two youth 
coordinators and five of the six youth advisors. One youth advisor was unable to participate in 
the interviews due to prior commitments. 

Interviews were tape recorded by the interviewer and transcribed by AEL support staff. 
Analysis was then conducted by question and theme, such that themes were identified within 
replies to each question and then tabulated. In other words, answers to each question were 
grouped thematically. 

Client Surveys 

Two surveys were developed by the evaluator to assess client satisfaction with various 
components and characteristics of the RESA I youth services program. Drafts of both surveys 
were provided to key youth services staff for feedback in March 2002; both were approved by 
RESA I staff in early April. 

In-School Youth Survey. The survey measuring the satisfaction of in-school youth with 
the RESA I youth services program consists of 17 closed-response option items and 2 open- 
ended items. The first question asks respondents to place a check mark next to brief descriptions 
of program services they may have received. Items 2 through 4 are the three customer satisfaction 
questions mandated by the U.S. Department of Labor. The three items provide 10-point scales 
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with which respondents are asked to rate their responses. Items 5 through 17 are questions 
designed to ascertain more specific information about youth’s experience with the program. 
Response options range from 1 (strongly disagree) to 4 (strongly agree). Finally, the last two 
questions seek more in-depth, qualitative information about youth’s suggestions for program 
improvement and need for further assistance. 

An informed consent form for parents or guardians of children under the age of 18 was 
developed for this administration of the in-school youth survey. The form, attached to the survey 
itself, asked parents or guardians of potential respondents to indicate their permission for the 
child’s participation in the survey process. The form briefly explained the purpose of the survey 
and assured parents/guardians of their child’s anonymity. 

A proportional random sample was drawn from the population of RESA I in-school youth 
participants to receive the survey. In other words, county sample sizes are proportional to county 
participant population sizes vis-a-vis the overall RESA I in-school participation population. 

The overall sample size was determined using the Creative Research Systems (CRS) 
online sample size calculator. To achieve a 95% confidence level, with a confidence interval of 5 
points, a sample of 257 would need to be drawn from the overall population of 774. A 
confidence level “is expressed as a percentage and represents how often the true percentage of 
the population who would pick an answer lies within the confidence interval” (CRS, 2001). In 
other words, with a sample size of 257, RESA I staff can be reasonably certain that the sample 
findings represent the population. 

The sample sizes per county are presented in the table below. 



Table 3 

Proportional County Samples Sizes 



RESA I County 


County In-school 
Participant Population 


Percent of Overall 
RESA I Participant 
Population 


Proportional Sample 
Size 


Fayette 


105 


13.57 % 


35 


Greenbrier 


74 


9.56 % 


24 


McDowell 


76 


9.82 % 


25 


Mercer 


130 


16.80 % 


43 


Monroe 


20 


2.58 % 


7 


Nicholas 


86 


11.11 % 


29 


Pocahontas 


16 


2.07 % 


5 


Raleigh 


86 


11.11 % 


29 


Summers 


30 


3.88 % 


10 


Webster 


63 


8.14% 


21 


Wyoming 


88 


11.37 % 


29 


TOTAL 


774 


100.00 % 


257 
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Surveys were sent to the RESA I Youth Services office at the end of April. The surveys 
and consent forms were then distributed to the sampled students by each county’s youth advisor 
in person beginning May 1. A reminder letter and additional copies of the survey and consent 
form were mailed to sampled students for whom RESA I had addresses who had not yet returned 
comoleted su r vp.vs rN=2H on Mav 21. 

Out-of-School Youth Survey. The survey assessing the satisfaction of out-of-school 
youth was designed to be conducted via telephone. The instrument is identical to the in-school 
youth survey, with the exception that it consists of a more elaborate protocol for use by telephone 
interviewers. Thus, the survey contains 17 forced-choice items and 2 open-ended items parallel 
to those on the in-school youth survey. 

Informed parental consent was not required of the out-of-school youth respondents. This 
is because out-of-school youth are over 18 years of age. 

RESA I staff and the evaluator chose to administer the survey to the total population of 
out-of-school students given its relatively small size of 1 14. Eight of the 1 14 had no telephone 
contact information, reducing the number of out-of-school youth the evaluation team attempted 
to contact to 106. 

A trained assistant began conducting the telephone surveys during the week of May 6, 
2002. Due to the large number of sampled out-of-school youth who were not available during the 
daytime at the provided phone numbers, the evaluator and assistant conducted telephone surveys 
in the evenings beginning the week of May 21. 

Employer Survey 

A survey measuring employer satisfaction was developed by the evaluator in March 2002 
and approved by pertinent RESA I staff in early April. This instrument, designed to be 
administered over the telephone, includes 15 forced-choice items and 2 open-ended prompts. The 
first item asks respondents to identify whether they provided unpaid or paid work experiences (or 
both) to program youth. The next three items are the three Department of Labor mandated 
satisfaction questions, with 10-point scales. Items 5 through 15 request employers to rate their 
level of agreement with various statements about their program experiences using a 4-point scale, 
in which 1 indicates strong disagreement and 4 indicates strong agreement. The final two items 
request suggestions for improving employer participation and a list of job skills employers think 
are needed by youth beginning program work experiences. 

As with the out-of-school youth, informed consent was not deemed necessary. Employers 
were assumed to be adults over 18 participating in one instance of anonymous and confidential 
data collection. 
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As with the out-of-school youth participants, the evaluator chose to administer the survey 
to the population of employers who collaborated with RESA I during the contract year of June 
2001-May 2002. The small population size in the program’s first year of operation made such a 
decision feasible. 

The survey was administered via telephone by a trained research assistant beginning the 
week of April 29 and ending the week of May 20, 2002. 

Database Review 

During the spring of 2002, RESA I youth services staff developed, in collaboration with a 
programmer, a database for the storage and analysis of ISS information. Ultimately, the database 
will replace the ISS, serve as a repository for all relevant case information, and provide reporting 
capabilities for staff and the evaluator. 

However, at the time of this writing, the reporting functions of the database were not yet 
in place. Moreover, no program customers had yet exited the program, making the calculation of 
performance objectives impossible. Database review, thus, will take place at the close of the 
program’s second year of implementation, June 2002-May 2003. 

Document Review 

Document review consisted almost entirely of the collection and analysis of media 
coverage of the Governor’s Workforce Investment Office and the state Workforce Investment 
Board, and federal documents pertaining to WIA regulations. Census data for the 1 1 Region I 
counties were also accessed. These analyses provided the evaluator with information about the 
context within which the RESA I youth services program operated in its first year. 

The evaluator also reviewed early drafts of the ISS and participated in a staff training 
session on, among other topics, the development of an effective, comprehensive, and useful ISS. 
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FINDINGS 

In this section, formative findings from the interim staff interviews are reported. 

Customer and employer satisfaction measures are also discussed. 

Interim Staff Interview Findings 

The following account describes findings from the staff interviews conducted in 
November 2001, approximately six months into the program’s implementation. These findings 
are also reported in the Interim Evaluation of the RESA I Workforce Investment Youth Services 
Program (Howley, 2002). 

Youth Advisor Interviews. Youth advisors reported a consistent account of the tasks 
they performed during a typical work day. These included traveling to school sites; meeting with 
students to develop their Individual Service Strategies (ISS) to set educational and career goals 
and to initiate service implementation; and offering “comprehensive guidance,” as one 
interviewee put it. 

Interviewees were asked what had most helped them conduct their work thus far. All five 
youth advisor respondents reported that other RESA I Youth Services staff, including their 
supervisors, had been the component of their experience proving most valuable in their efforts to 
implement the work mandated by the WLA. The team provided “tons and tons of support,” is 
how one interviewee put it. Said one youth advisor, “I brag all the time; I tell everybody, you 
know, I am surrounded by the smartest group of [people] that I know ... I have an excellent 
coordinator . . . [S]he gets things done that other people can’t get done . . . [The other advisors] 
kind of watch out for me.” Another interviewee noted the “camaraderie” among the youth 
advisor team. And yet another said, “The support of the other Youth advisors and the 
coordinators and the directors [has been the most helpful] . . . This is the most positive group of 
people I’ve ever worked with, and everybody’s just helpful. If you don’t know [something], you 
can call anybody at any given minute and they’ll help you work through it.” 

One interviewee clarified that the most helped team mode of operation was regular and 
collaborative staff meetings: “Everything we do is in the team. So, you know, we have the 
monthly staff meetings, and we all get together to develop [things]. Everything is done together. 
So when we develop a new form and one of us has a new idea, [we say,] ‘Hey, do you think this 
would work for you?’ And then we all use it together.” As a result of such open collaboration, 
this youth advisor reported, “I don’t think I’ve had a negative day here since we’ve been here. It’s 
amazing . . . Even if your idea is shot down, it’s shot down in a good way, you know, so that you 
don’t feel stupid.” 

Respondents were then asked to describe two or three additional things that might help 
them better conduct their work. In reply, all youth advisors with responsibility for more than one 
county school district (four of the five interviewees) said that they would prefer to work with 
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only one district apiece. Interviewees suggested that they would be able to provide more 
comprehensive, in-depth, and quality service to youth if they had smaller caseloads and less 
traveling to and from district sites. Lightened caseloads would necessitate hiring more youth 
advisors to accommodate all the clients, but staff maintained during interviewees that advisors 
would be better equipped io offer guidance and consistent follow-up under such circumstances. 



One staff member phrased it this way: “If we only had one county, I would feel like I 
could give those students a little more time and stop back and see them in a day or two, 
where[as] now I just don’t know where I’m going to be and I’m running all over.” Another said 
that travel “takes up a lot of time when I could be with the kids doing stuff.” 



Interviewees only made a few additional suggestions in terms of strategies that would 
help them conduct their work more successfully. Two advisors reported that computer training 
would be beneficial to them, although neither specified the type of training in which they were 
interested. 



Two interviewees suggested that the provision of a company vehicle might be useful. 
Both, however, indicated that this was not a priority issue; in fact, one acknowledged that such a 
request was probably an item on a “wish list.” (This advisor went on to acknowledge that staff 
had been provided ample tools to conduct their work, noting that their supervisors “have treated 
us well. We have everything we need . . . We have laptops, we have [software] programs, we 
have cell phones, we have voice mail now . . . We have a great secretary.”) 

Two staff members noted that the process for providing services to youth and for data 
collection, the ISS, was being finalized. Becoming acclimated to the process and converting the 
ISS to a computer database would be helpful to their work, according to both advisors. Both also 
noted that this was not an area of concern because they understood that the ISS and the ISS 
database were works in progress. As one reported, “We’re getting everything into the computer 
so it will be one system that flows [with] so much less paperwork . . . But that’s being developed, 
so that’s been taken care of.” 

According to one youth advisor, training in the interpretation of student assessment data 
would be helpful. This advisor also suggested that RESA I Youth Services staff convene a panel 
of business people to articulate their expectations for youth employees; making such standards 
explicit would help youth advisors prepare their clients more effectively, and would provide 
youth a means of learning what potential employers seek in applicants. 

Youth advisors were asked to describe what strategies had been most effective in helping 
their clients, primarily out-of-school youth, meet career goals. However, because advisors were 
still in the process of completing ISS’s with youth, interviewees had little to say in response to 
this question. “I only have, like, three out-of-school students that are still in the referral stage, so I 
don’t get to focus on the actual getting them a job,” reported one advisor, for example. 
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Nonetheless, interviewees did note a few strategies they found effective in helping 
students meet their career goals. Two advisors reported that helping students identify their career 
clusters and become more involved in the career exploration opportunities offered at their 
schools had been useful strategies. Both additionally indicated that many students were not even 
aware of the career cluster to which they belonged and which, in part, structured what courses 
they were assigned. For instance, one interviewee said, “I have seniors that don’t know what their 
career cluster is, and this is what all their classes are based on.” 

Two other advisors reported that consistent, and sometimes aggressive, follow-up was 
effective. One interviewee recounted situations in which s/he had taken youth to work sites when 
they had otherwise planned not to attend, for instance. The other advisor added that follow-up 
included collaboration with the Human Resource Development Foundation (HRDF), an 
organization providing some services to out-of-school youth. 

Staff were also asked what strategies had been less effective in helping youth achieve 
their career goals. One reported that “slow Job Service turnover was a serious concern. I 
completed applications with them [youth] in August, and this is November, and they’re still not 
up and going,” the advisor said. Although the youth coordinator supervising this advisor worked 
to facilitate faster turnover, the issue of attempting to collaborate with a bureaucratic 
organization remained and thereby continued to impede advisors’ efforts. 

According to another interviewee, the infrequency with which advisors were able to meet 
with each participant posed a challenge to efforts to assist students in achieving their career 
goals. As s/he reported, “They don’t really understand when I sit down to go over the program 
with them. It’s just all new. They’re like, ‘What? What are you talking about?’ And, again, our 
numbers come into play. I’m going to see a student, possibly, four times a year. So how much 
work can I do with a student for fifteen minutes [four times a year] ... an hour a year?” 

One respondent thought that the ongoing changes to the ISS were less than helpful. 
Nonetheless, the advisor also understood that such changes were an unavoidable part of the ISS 
development process. 

Next, youth advisors were asked to describe what strategies were proving most effective 
in helping clients meet their education goals. Four of the five youth advisors participating in the 
interviews reported that the counties’ after-school and tutoring services seemed to be very 
successful. One advisor put it this way, “All of the actual staff that the county has hired to work 
with individual groups are great. They will bend over backwards for these kids . . . They have 
one at each school that provides the after-school program, and they offer tutoring, study skills, 
whatever. But the staff, they have just fallen for these kids.” The county-level staff, in fact, had 
been so successful that “they actually have kids that want tutoring . . . The kids are saying, ‘I 
want tutoring. Yeah, I want help. I want to raise this English grade or this science grade . . . 
That’s amazing, because I know when I was in school, heavens no, I didn’t want any more 
school.” 
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Interviewees also mentioned the incentives for participation in the program that some 
counties had implemented. In one county, for example, “every six weeks, if they’ve got good 
attendance of some sort, they get $100. That’s one strategy I like.” Other counties provided 
opportunities for students to perform work and earn spending money, or to acquire leadership 
skills. Another county was planning to provide bus transportation for students participating in 
after-school Youth Services activities to facilitate their attendance. 

Two youth advisors noted their own strategy for helping clients reach their education 
goals. Both reported that they assisted students in articulating what they hoped to achieve in 
terms of academics. They then asked students to explicate the steps needed to meet their goals. 
One advisor explained, “That’s what I talk to them about. ‘Your goal is to graduate high school. 
But what’s the first step? You know, we’ve got to get you out of science class.’ Then we work on 
it that way ... I think that approach helps them. They see progress, and it helps the bigger picture 
become clear to them.” 

One interviewee thought that the paid summer work experience provided to youth had 
been a useful strategy to help clients meet their education goals. Although the advisor noted that 
this approach was being “phased out,” s/he nonetheless reported that it had been a valuable 
opportunity for participants. 

Asked what had been less helpful in assisting clients to meet their education goals, only 
two youth advisors responded with concerns. One interviewee reported difficulties with ensuring 
that the county-level coordinators offered all the services required: “They say, ‘Don’t make 
promises to these kids that you can’t keep,’ when we clearly have a list of [program] elements 
that we offer.” For this advisor, the county was the impediment to full implementation. 

The second respondent reported similar issues with the county-level program. “You’d 
have to have angels for a program like that to work,” s/he said. This staff member explained that 
the program at the county level was not structured to encourage and support participation of 
struggling students. For example, incentives for participation in after-school activities were not 
offered. Even more troubling to the advisor was the lack of transportation from after-school 
activities to students’ homes in this rural, impoverished county. According to the advisor, county 
staff should offer all the support they could afford in order to assist students; this not being the 
case, s/he suspected they were not as committed to improving the circumstances of students as 
might be desired. 

Youth advisors were asked to make recommendations to improve services to youth. Three 
of the five respondents recommended that more advisors be hired so that each team member is 
ultimately responsible for fewer students and fewer counties. Said one interviewee, “If I were in 
one county, I could be building stronger relationships faster.” 

One interviewee recommended that counties be encouraged to provide transportation to 
students, both from after-school activities and to enrichment opportunities outside the county 
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borders. Another youth advisor suggested that professional marketing tools be developed to 
advertise the program. According to another respondent, more, and more effective, 
communication among staff would be helpful. And one interviewee recommended that staff 
simply continue to nurture relationships with school staff and students. 

Youth Coordinator Interviews. Youth coordinators reported completing a variety of 
tasks in the conduct of their work. These included communicating with and providing oversight 
of youth advisors; providing information about the program to local civic, judicial, social, and 
education representatives; meeting with county coordinators to discuss budget and program 
issues; and facilitating collaboration between agencies. 

Asked to describe any major accomplishments of the RESA I Youth Services program, 
both youth coordinators noted that the ISS process and template had been developed and that the 
majority of ISSs had been completed. As one put it, “The biggest thing that really started causing 
things to gel was developing an ISS form that we could live with that was practical, and then 
sitting down and actually doing the ISSs.” Both coordinators estimated that approximately 80% 
of ISSs had been completed at the time of the interviews. This, in their view, was an especial 
accomplishment because, as one explained, “The counties were given their money and told to go 
through with this program. And they really had a month and a half jump on us before we came on 
board. So we played catch up for the first couple of months.” 

Interviewees mentioned other accomplishments as well. One mentioned that the 
facilitation of a summer youth program in each of the 1 1 counties had been successful, 
particularly as it was conducted in the midst of hiring new staff and developing the new WIA 
program. The other coordinator reported that RESA I staff had attended a summit in Philadelphia 
for WIA programs in the mid- Atlantic region. He explained, “We came away from there 
knowing that we’re doing pretty much what we should be doing.” The summit experience, then, 
validated the efforts of staff. 

Youth coordinators were next asked what activities had yet to be completed. Both 
reported that approximately 20% of ISSs remained to be completed. One coordinator additionally 
noted that the program was required to locate and invite about 1,200 former JTPA summer youth 
program attendees to participate in the RESA I Youth Services program. Because many students 
had moved, transferred, graduated, or left school, the coordinators found this task very 
challenging. 

Coordinators described several difficult issues they had confronted thus far. One 
interviewee reported that “catching up on the ISS was the hardest.” As he explained, this 
challenge was compounded by the difficulties of advisor responsibility for several districts and 
by school schedules: “Well, you know, they have multiple counties . . . and it takes a while when 
you have to go out to the school system and work with the high school schedule or the junior 
high schedule. When you can pull this one student out of class, you don’t want to-they’re already 
behind, a lot of them. You don’t want them to miss. So you may go out to a school and be able to 
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do one or two students in the morning, one or two at lunch, and hopefully you can pull a few out 
through the day through their free periods, or their advisory periods. We don’t want to puil them 
out of math or English when they’re already struggling. So that’s a long time, to figure that you 
get three or four a day, and you have a hundred [total]. We hope to finish by next year.” 

This coordinator also noted that some of the school principals were resistant to the 
program given the many other programs competing for their attention: “The principals didn’t 
really want to come on board because so many are coming to their schools anyway . . . [There 
are] a million programs out there, and we’re just one trying to fight for a piece of time. 

The other coordinator reported that the most difficult issue staff confronted concerned the 
large caseloads advisors had been given. The challenge of meeting the needs of a large caseload 
was compounded by the amount of time required to travel to and from counties. 

Both coordinators were then asked to make recommendations to improve services to 
youth. One suggested that the eligibility process be streamlined. Asked to clarify this, he 
continued, “If a student is interested in the program, they will call Job Service or just show up at 
Job Service and in some counties . . . there’s no Job Service there so you need to travel to 
Beckley or you need to travel to Welch. For kids, who don’t have any money anyway and 
parents may not have any transportation, that’s tough. And once you get to Job Service, you have 
to register . . . and there’s probably 20 forms of documentation you have to have-proof of 
residence, birth certificate, proof of income, citizenship, selective service, just so many things for 
a kid who . . . doesn’t find things easy or nothing comes easy for them and here they have to 
work to make two or three trips backwards and forth, bringing in documentation. Uh, in my 
experience in working with at risk youth, they’ll make an effort once, maybe twice, you know. If 
they don’t succeed, they’re not going to try again ... So unless you can catch them on that first 
trip through the doors, you’re going to lose them. And forms need to be simple where someone 
on a 5th or 6th grade reading level can complete them, because most of their parents are on the 
same level, if that high.” The other coordinator reiterated this point, saying, “A lot of the times, 
because of the nature of the clients, it’s difficult to get them to fill out the paperwork..” 

One coordinator also mentioned a difficulty with the satellite centers. He explained, “In 
some of the rural counties, the one-stop centers are satellite centers. In other words, there’s no 
one on staff on a regular basis to do the testing. For example, in Pocahontas County, the person 
that does the testing has to drive from Lewisburg. And it’s hard, and it’s understandable, [he or 
she] just can’t come up here and test one person. [He or she] has to wait until she gets five or six 
or a certain number, and then [he or she] comes up for testing. And that’s difficult to get all of 
these youngsters together and again you have to understand, our client, clientele is the reason that 
they need to be in the program is because they have problems with schedules and transportation. 

In addition, one of the coordinators suggested that services to clients would be improved 
if each county contact for RESA I Youth Services were not also responsible for a variety of other 
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programs. This interviewee added that such an arrangement might not be financially feasible, but 
he nonetheless thought it would be one strategy to strengthen the program. 

The youth coordinators were asked to what they would recommend to make their own 
jobs easier. One coordinator replied that he was very satisfied with the support he received: “This 
is great.” The other coordinator likewise reported that he did not require anything further to 
improve his work conditions. However, he again suggested that youth advisors have smaller 
caseloads. 

Finally, the coordinators were asked to provide any additional information that might be 
important to the evaluation. One respondent praised the staff and the organizational culture: 
“They’re just, they support us in every way that you could, but at the same time they give us our 
freedom to experiment.” The coordinator went on to express appreciation for the way in which 
the program structure accommodated the particular rural needs of the area: “What s unique about 
this program, unlike most federal programs . . . where guidelines come down that do not apply to 
rural settings, [to the] uniqueness of the community . . . with us having a local board that sits in 
each area, they’re able to . . . identify and understand the uniqueness of the community and the 
needs of our potential clients, the needs of the work force, the businesses.” 

The other coordinator reported a concern with the enrollment process. He noted that, 
upon enrollment, youth were to be enrolled in follow-up activities, although such activity could 
not take place until youth exit the program. The coordinator suggested that this issue be resolved. 

Customer Satisfaction Survey Findings 

Customer satisfaction findings from the three client surveys are discussed below. 
Response rates, descriptive statistics, and calculations mandated by the Department of Labor for 
all WIA-funded programs are presented as well. 

In-School Youth Satisfaction Survey Findings. Of the participating in-school youth 
receiving RESA I services this year (N=774), 257 were selected for inclusion in the satisfaction 
survey using the Creative Research Systems (CRS) online sample size calculator (see p. 11 for 
more details). Unfortunately, the response rate for this group of youth was very low. (This is not 
necessarily surprising, however, given the method by which surveys were distributed, the need 
for signed parental consent forms, and the youth of the participants.) A total of 60 youth returned 
completed questionnaires, representing a return rate of 23%. Such a low rate of response renders 
the generalizability of findings to the larger RESA I in-school youth population very tenuous. 

Table 4 includes the number and percentage of respondents per RESA I County. Mercer 
County had the largest percentage of respondents, while there were no respondents from 
McDowell and Wyoming Counties. All county data are reported together; individual county 
responses were too low for disaggregation of survey data to yield meaningful results. 
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Table 4 

Number and Percentage of In-School Youth Surveyed by County 



22 



County 


N 


Percentage 


Fayette 


11 


18.3 


Greenbrier 


4 


6.7 


McDowell 


0 


0.0 


Mercer 


19 


31.7 


Monroe 


2 


3.3 


Nicholas 


10 


16.7 


Pocahontas 


2 


3.3 


Raleigh 


4 


6.7 


Summers 


3 


5.0 


Webster 


5 


8.3 


Wyoming 


0 


0.0 



The survey requested that the respondents specify all of the RESA I services they 
received. Most reported that they received more than one service. The majority of respondents 
indicated that they participated in summer employment opportunities (68.3%) and paid or unpaid 
work experiences (63.3%). More than half of the respondents (53.3%) reported that they 
received job skills training. The least utilized services were alternative high school (5.0%), child 
care or transportation (10.0%), mentoring (15.0%) and follow-up services (15.0%). These data 
are presented in Table 5. 

Internal consistency analyses were conducted to measure the extent to which satisfaction 
survey items measure a unitary construct. Overall, the instrument possessed sufficient internal 
consistency reliability for this administration, with a Cronbach’s alpha coefficient of .88. The 
three federally mandated items, analyzed as a subscale, were quite reliable, with an alpha of .92. 
Finally, with a Cronbach’s alpha of .89, the remaining satisfaction items also possessed adequate 
internal consistency reliability for this administration. 

Results from the three federally mandated items administered to in-school youth 
respondents are presented in Table 6. The mean ratings ranged from 7.71 to 7.59 on the 
Department of Labor mandated 10-point scale. In-school youth satisfaction was also measured 
as the percentage of respondents who gave an average rating of 5.6 or higher on the 10-point 
scale across the three federally mandated satisfaction questions. For the purposes of this rating, 
5.5 represents a satisfaction assessment of “average.” This process was done to assess the literal 
meaning of the RESA I satisfaction goal. RESA I exceeded the performance objective for in- 
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house youth satisfaction, with the percentage of respondents rating their satisfaction at above 
average ranging from 86.4% to 75.0% This is well above the 69.0% performance level set by 
RESA I in agreement with the state WIA board. 
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Number and Percentage of In-School Respondents Using Type of Service 



Service 


N 


Percentage 


Alternative high school 


3 


5.0 


Child care or transportation 


6 


10.0 


Follow-up services 


9 


15.0 


Mentoring 


9 


15.0 


Leadership training 


17 


28.3 


Guidance and counseling 


17 


28.3 


Tutoring or study skill training 


20 


33.3 


Job skills training 


32 


53.3 


Paid or unpaid work experiences . 


38 


63.3 


Summer employment opportunities 


41 


68.3 



Table 6 

Federally Mandated Items for In-School Youth 



Item 


N 


Mean 


SD 


% over 5.6 


Overall satisfaction 


59 


7.71 


2.21 


83.10* 


Expectations 


59 


7.64 


2.29 


74.60* 


Ideal program 


59 


7.59 


2.33 


83.10* 



♦Meets federal WIA performance objective requirement. 



The mean ratings for each of the three federally mandated items were converted to a 100- 
point scale in order to calculate in-school youth satisfaction conforming to the ACSI from which 
the items are taken. These values were then multiplied by the federally approved state formula 
weights and then added. The calculations, which resulted in an ACSI score of 73.95, are shown 
as follows: 

Overall satisfaction: (7.7 1-1 )/9 x 100 = 74.56 
Expectations: (7.64-l)/9x 100 = 73.78 

Ideal program: (7.59- 1)/9 x 100 = 73.22 

(74.56 x .4092) + (73.78 x .3283) + (73.22 x .2625) = 30.51 + 24.22 + 19.22 = 73.95 

eric 
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The responses to the twelve questions asked to ascertain information about the 
respondents’ experience with the program are summarized in Table 7. The mean ratings ranged 
from 3.66 to 2.87 on the 4-point scale developed by the evaluator and program staff. The 
standard deviations ranged from 0.58 to 0.85. The majority of respondents (94.6%) agreed that 
they were glad they had participated in the program (3.66, SD 0,58). The small standard 
deviation is evidence that there is little variation in the respondents’ level of agreement with this 
item. Most of the respondents (92.9%) either agreed or strongly agreed that the staff were 
knowledgeable (3.50, SD 0.69) and that the staff respected them (3.46, SD 0.79). However, the 
somewhat higher standard deviations of 0.69 and 0.79 show that there was more variation in the 
respondents’ agreement concerning these items. In addition, 89.3% of the respondents either 
agreed or strongly agreed that the staff were helpful (3.52, SD 0.69) and 87.5% felt that the staff 
helped them find ways to participate (3.20, SD 0.75). It is also worth noting that the majority of 
respondents agreed or strongly agreed that they did not have to wait a long time to receive the 
services they needed (85.7%), received a lot of information through the program (83.9%), and 
were involved in choosing the services that were right for them (81.8%). 



Table 7 



Descriptive Statistics for In-School Youth Satisfaction II 


terns 


Item 


N 


Percent 

agree 


Percent 

disagree 


Mean 


SD 


The services I received helped me meet my 
education goals. 


56 


76.8 


23.2 


2.96 


0.76 


The services I received helped me meet my 
career goals. 


56 


69.6 


30.4 


2.87 


0.83 


I was involved in choosing the services that are 
right for me. 


55 


81.8 


18.2 


3.11 


0.79 


The staff of this program were helpful. 


56 


89.3 


10.7 


3.52 


0.69 


The staff of this program were knowledgeable. 


56 


92.9 


7.1 


3.50 


0.69 


I felt that the staff respected me. 


56 


92.9 


7.1 


3.46 


0.79 


I did not have to wait a long time to receive the 
services I needed. 


56 


85.7 


14.3 


3.21 


0.73 


I received a lot of information through this 
program. 


56 


83.9 


16.1 


3.07 


0.83 


Staff helped me find ways to participate in the 
program even if there were circumstances that 
made doing so difficult for me. 


56 


87.5 


12.5 


3.20 


0.75 


I had experiences in this program that I would 
not have had otherwise. 


56 


78.6 


21.4 


3.16 


0.85 


I feel more prepared for my future because of 
this program. 


56 


80.4 


19.6 


3.21 


0.80 


I am glad I participated in this program. 


56 


94.6 


5.4 


3.66 


0.58 



O 
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The least highly rated item, with a mean of 2.87 (SD 0.83), was that the program helped 
the respondents meet their career goals. The extent to which the program helped the respondents 
meet their education goals was also less highly rated. However, with a mean of 2.96 (SD 0.76), 
many of the respondents did agree that their participation in the program helped them achieve 
their education goals. Further, 21.4% (3.16, SD 0.85) of respondents disagreed that they had 
experiences in this program that they would not have had otherwise. 

In-school youth were also asked to complete two open-ended prompts in the final section 
of the survey. The first of these, If I could change two things about the program, I would change 
. . ., received replies from 48 respondents. Of these, 25 provided answers with multiple themes. 
Fifteen replies indicated that time was an issue: eleven respondents wanted more work hours, one 
fewer hours, two a longer summer program, and one more time for tutoring and community 
work. Thirteen responses suggested that participants be compensated with higher wages for their 
work, and one of these further suggested that participants be paid weekly rather than at the end of 
the summer. More activities, trips, and meetings were requested by nine in-school youth, and six 
others recommended that a greater variety of job types and opportunities be offered to 
participants, including job shadowing and specific job skills. Two responses requested more 
information from teachers; one of these suggested that teachers provide more information on 
college, specifically. Two respondents sought more assistance with transportation. Fifteen 
respondents reported that they would change nothing about the program. 

The remaining themes were idiosyncratic and therefore could not be categorized. 
According to these themes, changes should be made to the program rules, the tools with which 
students perform their work, the food provided, the age at which participants are able to be 
employed, and the speed with which pay checks are received. One respondent requested that 
work sites be changed. Other suggestions included more assistance from teachers with job 
seeking, more polite staff, and enhanced general assistance from staff. One respondent also 
requested that students not have to work “in the rain.” 

The second open-ended prompt, At this point in the program, I could use some help with . 

. . , received 33 replies. The remaining respondents either did not complete the prompt or 
reported that they did not need help with anything. Nine respondents wrote that they wanted 
assistance with particular academic skills: three sought assistance with reading, two with math 
and science, one with geometry, one with math only, one with math and reading and test-taking, 
and one with social studies. Five respondents indicated that they would like help with finding 
work. Four requested further assistance with planning for post-secondary experiences, whether 
work, training, or college. Four respondents sought improvement of their social skills, including 
public speaking and controlling their frustration. Assistance with homework was requested by 
two youth. Two youth sought more work experience. 

The remaining responses were unique and could not be categorized into themes. One 
respondent requested follow-up services, while another reported wanting assistance with 
“everything.” Learning to become a better employee was important to one respondent, whereas 
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another simply requested the opportunity to do “more things.” Preparation for an upcoming 
military examination was sought by one respondent, and another simply requested more 
involvement with the program. One student sought more information about nursing; another, 
more generally, requested assistance with choosing a career. 

Out-of-School Youth Satisfaction Survey Findings. All participating out-of-school 
youth receiving RESA I services this year (N=l 14) were selected for inclusion in the satisfaction 
telephone survey. Of the 1 14, 39 were unavailable because they were no longer at the provided 
phone number and no new number was supplied; their phone service was disconnected; or the 
number on record was incorrect. These circumstances reduced the available sample to a total of 
75. Ultimately, 31 out-of-school youth completed the telephone survey, for a response rate of 
41%. 



The number and percentage of respondents per RESA I county are presented in the 
following table. The largest percentage of respondents resided in Raleigh County, and there were 
no respondents hailing from Monroe County. County responses were too low, however, for 
disaggregation of survey data to render meaningful results; thus, all county data are reported 
together. 



Table 8 

Number and Percentage of Out-of-School Youth Surveyed by County 



County 


N 


Percentage 


Fayette 


5 


16.1 


Greenbrier 


1 


3.2 


McDowell 


3 


9.7 


Mercer 


2 


6.5 


Monroe 


0 


0.0 


Nicholas 


2 


6.5 


Pocahontas 


4 


12.9 


Raleigh 


8 


25.8 


Summers 


3 


9.7 


Webster 


2 


6.5 


Wyoming 


1 


3.2 



Of the total 31 who agreed to take part in the telephone survey, eight reported that they 
had not participated in the RESA I program or did not know anything about it. These respondents 
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constitute slightly more than a quarter (26%) of the sample. Because they were therefore unable 
to respond to survey items, they are excluded from the analyses presented below. 

Respondents were asked to indicate all of the RESA I services they had received. Many 
reported that they had received more than one service. The most oft-received service was the 
provision of job skills training; 87.0% of respondents indicated participating in such a service. 
Nearly two thirds (65.2%) of respondents each had participated in paid or unpaid work 
experiences or had taken advantage of summer employment opportunities. The service least 
accessed was child care or transportation (30.4%). These data are presented in Table 9. 



Table 9 

Number and Percentage of Out-of-School Respondents Using Type of Service 



Service 


N 


Percentage 


Child care or transportation 


7 


30.4 


Leadership training 


8 


34.8 


Follow-up services 


10 


43.5 


Guidance and counseling 


10 


43.5 


Alternative high school 


10 


43.5 


Mentoring 


11 


47.8 


Tutoring or study skill training 


13 


56.5 


Paid or unpaid work experiences 


15 


65.2 


Summer employment opportunities 


15 


65.2 


Job skills training 


20 


87.0 



To determine the extent to which satisfaction survey items measure a unitary construct, 
internal consistency reliability analyses were conducted. With a Cronbach’s alpha of .88, the 
overall instrument possessed sufficient reliability for this administration. The three federally 
mandated items possessed much less satisfactory internal consistency reliability, with an alpha 
coefficient of .55. Finally, the remaining 10 quantitative items possessed very adequate 
reliability, with a Cronbach’s alpha of .93. Overall, then, this instrument is reliable for this 
administration, but interpretation of the three federally mandated items as a subscale assessing a 
unitary construct should be conducted with caution. 

Results from the three federally mandated items are shown below. Mean ratings ranged 
from 8.48 to 6.63 on the 10-point scale. To address the literal meaning of the RESA I satisfaction 
goal, out-of-school customer satisfaction was also measured as the percentage of respondents 
who gave an average rating of 5.6 (where 5.5 represents a satisfaction assessment of “average”) 
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or higher on the 10-point scale across the three federally mandated satisfaction questions. Using 
this measure, RESA I exceeded the performance objective for out-of-school youth satisfaction, 
with the percentage of respondents rating their satisfaction above average ranging from 73.7% to 
92.5%. This is well above the 69.0% performance level. 

Table 10 

Federally Mandated Items for Out-of-School Youth 



Item 


N 


Mean 


SD 


% over 5.6 


Overall satisfaction 


21 


8.48 


1.25 


92.5* 


Expectations 


20 


7.80 


2.35 


80.0* 


Ideal program 


19 


6.63 


3.18 


73.7* 



* Meets federal WIA performance objective requirement. 



To calculate out-of-school youth satisfaction according to the ACSI, from which the three 
federally mandated satisfaction items are drawn, the mean ratings for each of the three items 
were converted to a 100-point scale. These values were then multiplied by the federally approved 
state formula weights, and then added. These calculations, resulting in an ACSI score of 75.24, 
are shown below: 

Overall satisfaction: (8.48 - l)/9 x 100 = 83.11 
Expectations: (7.80 - l)/9 x 100 = 75.56 

Ideal program: (6.63 - l)/9 x 100 = 62.56 

(83.11 x .4092) + (75.56 x .3283) + (62.56 x .2625) = 34.01 +24.81 + 16.42 = 75.24 

Descriptive statistics for the remaining satisfaction items are displayed in the.following 
table. Mean ratings ranged from 3.65 to 3.10 on the 4-point scale, and standard deviations ranged 
from 0.49 to 0.94. Respondents were unanimous in their agreement (100.0%) that program staff 
were knowledgeable (3.62, SD 0.50) and that they had received a lot of information through the 
program (3.64, SD 0.49). The small standard deviations indicate that respondents diverged very 
little in terms of the degree of agreement with these items. Most respondents (95.5%) agreed or 
strongly agreed that staff has been helpful, with a mean rating of 3.59 (SD 0.59). 

Relatively least highly rated were the extent to which respondents thought that the 
program had provided them experiences they would not have otherwise had. However, with a 
mean of 3.10 (SD 0.83), most respondents (81.0%) agreed that they had, in fact, been provided 
such experiences through the program. With a mean rating of 3.15 (SD 0.75), respondents were 
also slightly less in agreement that the services they had received helped them meet their career 
goals: 20% disagreed that their career goals had been furthered via program participation. 
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Table 11 

Descriptive Statistics for Out-of-School Youth Satisfaction Items 
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Item 


N 


Percent 

agree 


Percent 

disagree 


Mean 


SD 


The services I received helped me meet my 
education goals. 


20 


85.0 


15.0 


3.20 


0.70 


The services I received helped me meet my 
career goals. 


20 


80.0 


20.0 


3.15 


0.75 


I was involved in choosing the services that are 
right for me. 


21 


85.7 


14.3 


3.38 


0.87 


The staff of this program were helpful 


22 


95.5 


4.5 


3.59 


0.59 


The staff of this program were knowledgeable. 


21 


100.0 


0.0 


3.62 


0.50 


I felt that the staff respected me. 


22 


95.5 


4.5 


3.52 


0.59 


I did not have to wait a long tome to receive the 
services I needed. 


21 


90.5 


9.5 


3.24 


0.77 


I received a lot of information through this 
program. 


20 


100.0 


0.0 


3.65 


0.49 


Staff helped me find way to participate in the 
program even if there were circumstances that 
made doing so difficult for me. 


20 


95.0 


5.0 


3.40 


0.75 


I had experiences in this program that I would 
not have had otherwise. 


21 


81.0 


19.0 


3.10 


0.83 


Staff were available to answer my questions. 


21 


89.5 


9.5 


3.29 


0.64 


I feel more prepared for my future because of 
this program. 


21 


85.7 


14.3 


3.24 


0.94 


I am glad I participated in this program. 


19 


94.7 


5.3 


3.58 


0.61 



Out-of-school youth were asked to respond to two open-ended prompts. The first of these, 
If I could change two things about the program, I would change . . ., received replies from 16 
respondents. Of these, seven provided answers containing two themes. Six respondents indicated 
that they would change nothing about the program because, as one put it, “the program is 
excellent.” Three responses concerned the instructional opportunities provided through the 
program; one respondent, for instance, requested more hands-on training. Four replies related to 
time: Two respondents requested “more hours,” another “less breaks,” and a fourth that the 
program would “last longer.” Enhanced participation and increased advertisement about the 
program was recommended by two other respondents. 
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The remaining replies were idiosyncratic and could not be categorized. One respondent 
wanted more pay, another more training and job opportunities, another more follow-up, and yet 
another less paperwork. Also proffered were suggestions that the program age limit be changed 
(although the respondent did not specify any further than this), that local programs be offered in 
closer proximity to students, and that some staff be more polite. Another respondent reported that 
the intake test had been a “surprise,” and that she or he hadn’t known to plan for it. 

The second open-ended prompt, At this point in the program, 1 could use some help with . 

. . , received 17 replies, only one of which contained more than one theme. The remaining 
respondents did not complete the prompt. Of those who did reply, five reported that they did not 
require further assistance. As one such participant phrased it, “They helped me with everything.” 
Four respondents wrote that they would like help finding a job. Two each requested assistance 
with obtaining a vehicle and improving their job skills. Planning for college was a theme cited by 
two participants, who wanted help both with planning for the undertaking and with information 
about financial aid. 

The four remaining replies were unique and could not be categorized. One youth sought 
help with science and social studies, another with money management, and a third with 
“checking] if my records are up to date.” The fourth respondent reported not knowing with what 
he or she needed help. 

Employer Satisfaction Survey Findings 

RESA I Workforce Development Youth Services staff supplied the names of 20 
employers who had participated thus far in the program by providing paid or unpaid work 
experiences to youth. Telephone information was unobtainable for two of the employers, 
decreasing the available sample to 18. One contact person no longer was employed at the 
organization involved with the RESA I program, further decreasing the available sample to 17. 

Evaluation staff could not reach five of the sample, and one respondent reported having 
no knowledge of the program. Ten employers participated in the telephone survey, and an 
eleventh refused to participate in the telephone administration but did complete the identical 
survey by mail. The eleven respondents constitute a response rate of 65%. 

Respondents were asked to indicate whether they had provided unpaid and/or paid work 
experiences to program clients. Most (81.8%) had provided unpaid work experiences, but 
somewhat more than a third (36.4%) had offered paid employment. 

To determine the extent to which satisfaction survey items measure a unitary construct, 
internal consistency reliability analyses were conducted. With a Cronbach’s alpha of .87, the 
overall instrument possessed sufficient reliability for this administration. The three federally 
mandated items possessed less satisfactory internal consistency reliability, however, with an 
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alpha coefficient of .69. Finally, the remaining 10 quantitative items possessed very adequate 
reliability, with a Cronbach’s alpha of .91. 

Descriptive statistics for the employer satisfaction survey’s three federally mandated 
questions are presented in Table 12. As shown, mean ratings ranged from 9.13 to 6.1 1 on the 10- 
point scale, and standard deviations ranged from 1.25 to 3.59. To address the literal meaning of 
the RESA I satisfaction goal, employer customer satisfaction was also measured as the 
percentage of respondents who gave an average rating of 5.6 (where 5.5 represents a satisfaction 
assessment of “average”) or higher on the 10-point scale across the three federally mandated 
satisfaction questions. Using this measure, RESA I exceeded the performance objective for 
employer satisfaction, with the percentage of employer respondents rating their satisfaction at 
above average ranging from 77.7% to 100%. This is well above the 67.0% performance level set. 

Table 12 

Federally Mandated Employer Satisfaction Items’ Descriptive Statistics 



Item 


N 


Mean 


SD 


% over 5.6 


Overall satisfaction 


8 


9.13 


1.25 


100.0* 


Expectations 


9 


6.89 


3.59 


77.7* 


Ideal program 


9 


6.11 


3.30 


77.7* 



* Meets federal WIA performance objective requirement. 



To calculate employer satisfaction according to the American Customer Satisfaction 
Index (ACSI), from which the three federally mandated satisfaction items are drawn, the mean 
ratings for each of the three items were converted to a 100-point scale. These values were then 
multiplied by the federally approved state formula weights, and then added. These calculations, 
resulting in an ACSI score of 73.32, are shown below: 

Overall satisfaction: (9.13 - l)/9 x 100 = 90.33 

Expectations: (6.89 - l)/9 x 100 = 65.44 

Ideal program: (6.11 - l)/9 x 100 = 56.7 

(90.33 x .4092) + (65.44 x .3283) + (56.70 x .2625) = 36.96 + 21.48 + 14.88 = 73.32 

Responding employers rated their level of agreement with statements about their 
satisfaction with various aspects of the program. As presented in the table below, mean ratings 
ranged from 3.80 to 2.82 on the 4-point scale. Employers agree most strongly that they would 
recommend the RESA I program to other employers (3.80, SD 0.42) and that they plan to 
continue participating in the program next year (3.55, SD 0.93). Respondents agreed 
unanimously (100.0%) that RESA I staff had been helpful. 
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On the other hand, employers were less satisfied that the process of providing work 
experiences to youth had been easy (2.82, SD 1.08), although the relatively large standard 
deviation suggests that there was some variability in respondents’ ratings. Employers were also 
somewhat less in agreement that the youth with whom they worked possessed the necessary 
skills for working in their organization (3.09, SD 1.04)-more than a quarter (27.3%) of 
respondents disagreed that youth possessed such skills. 



Table 13 

Employer Satisfaction Items’ Descriptive Statistics 



Item 


N 


Percent 

agree 


Percent 

disagree 


Mean 


SD 


The RESA I Youth Services staff communicated the 
purpose of the program to me clearly. 


9 


90.0 


10.0 


3.30 


0.66 


The RESA I Youth Services staff were helpful. 


9 


100.0 


0.0 


3.44 


0.53 


The RESA I Youth Services staff understood my 
needs as an employer. 


10 


80.0 


20.0 


3.20 


0.79 


When I had questions or concerns about the program, 
RESA I staff responded quickly to them. 


8 


87.5 


12.5 


3.50 


1.01 


I received adequate support from RESA I throughout 
the process of offering work experiences to youth. 


10 


90.0 


10.0 


3.20 


0.63 


The process of providing work experiences to youth 
has been easy. 


11 


72.7 


27.3 


2.82 


1.08 


The youth I worked with had the skills I thought they 
needed for employment in my organization. 


11 


72.7 


27.3 


3.09 


1.04 


The youth I worked with were well-behaved. 


11 


81.8 


18.2 


3.45. 


1.04 


I think experiences like the one(s) I provided will 
help youth in my area become better prepared for 
employment. 


10 


90.0 


10.0 


3.40 


0.70 


I would recommend this program to other employers. 


10 


100.0 


0.0 


3.80 


0.42 


I plan to continue participating in this program. 


11 


90.9 


9.1 


3.55 


0.93 



Respondents were asked to complete two open-ended phrases in the final section of the 
survey. The first open-ended question is as follows: Youth need to come to work experiences like 
the one(s) I provided with the following job skills . . . One employer gave six responses, four 
employers gave four responses each, four employers gave two responses each and two employers 
gave one response each. Embedded in the responses are several themes. Of these themes, seven 
related to the work ethic of participating youth, including promptness, ability to follow 
instructions, time management, and an overall good work ethic. Eleven of the themes related to 
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basic social skills, such as good manners, good behavior, respectfulness, and the ability to 
communicate well with others. Eight of the themes dealt with skill sets, including computer 
skills, organization skills, typing skills and basic skills such as good grammar, good math skills, 
and legible writing. Six of the themes represent qualities that cannot necessarily be taught in a 
classroom environment. They include self-confidence, common sense, a professional 
appearance, and a willingness to leam. 

The second open-ended question prompted respondents: It would be easier for me to 
participate in this program if. . . One employer gave three responses to this question, one 
employer gave two responses to this question, and nine employers gave one response each. 

Again, the themes embedded in each of the responses are delineated. Three of the responses 
contained the general theme that no changes were needed to render the program easier for 
employers. Another theme expressed in three other responses was the desire to know more about 
the program. Four of the responses contained a theme dealing with time. In other words, these 
employers would have preferred more time to devote to the program and more advance notice of 
when the youth would begin their work experience. Further, two employers voiced a desire to 
choose the time period for youths’ work experiences because of the seasonal nature of their 
businesses. Four themes related to the children themselves. One employer expressed a desire for 
contact with the children to be made easier. Another employer stated that the youth should be 
more well-behaved, should do the job that they are asked to do, and should be more closely 
supervised by RES A I staff. 
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CONCLUSIONS AND RECOMMENDATIONS 



A variety of conclusions may be made from the data collected over the course of the 
RESA I Youth Services program’s first year of implementation. In addition, the conclusions 
connote recommendations staff might consider as they continue to 
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Conclusions 

Conclusions are categorized by the stakeholder group to which they refer. 

Program Staff. Interim staff interviews indicate that staff feel supported and nurtured by 
their supervisors and colleagues alike. The program culture among youth advisors, youth 
coordinators, and the program director is collaborative and collegial. 

At the mid-point of the program’s first year, staff had few concerns. Most significantly, 
youth advisors would prefer to serve only one county apiece, an arrangement they reported would 
decrease caseloads and minimize travel time. Ultimately, staff thought that their services to 
clients would become more targeted, more intensive, and of better quality if they were 
responsible for fewer youth. According to staff in informal communications with the evaluator 
following the distribution of the Interim Evaluation of the RESA I Worlrforce Investment Youth 
Services Program , this concern was addressed by program administrators, who agreed to hire 
more youth advisors during the next contract year. 

Another challenge youth services staff faced was slow Job Service turnaround, a 
complication which effectively thwarted efforts to coordinate services to clients. Informal 
telephone communications with staff during April 2002 indicate that this remains a concern for 
RESA I Youth Services staff. 

In-School Youth. The survey response rate for this group of youth was very low, and no 
responses were received from two of the Region I counties. The generalizability of findings from 
this sample to the larger population of RESA I in-school youth participants is therefore 
compromised. 

In-school youth took most advantage of paid or unpaid work experiences and summer 
employment opportunities through the RESA I program. Many also received job skills training. 
Far fewer, however, availed themselves of alternative high school or child care and 
transportation. 

In-school respondents were satisfied overall with the RESA I program. Their expectations 
about the program tended to have been more than met, and the program approximated the ideal 
program for individuals in their circumstances. Respondents’ levels of satisfaction exceeded the 
federal performance objective. 
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Respondents were overwhelmingly glad to have participated in the program. Many also 
agreed that the RESA I staff were helpful, knowledgeable, and respectful of participants. On the 
other hand, in-school respondents were somewhat less in agreement that the program had helped 
them meet their career and education goals. 

In-school youth had need for both academic and work assistance. Tutoring in specific 
areas, preparation for their post-secondary lives, and assistance with job searches were requested 
often by respondents. Changes to the pay structure for work experiences were also sought by in- 
school youth. In addition, youth wanted increased work hours. 

Out-of-School Youth. The response rate for out-of-school youth was low. This is not 
altogether surprising, given that rural and/or impoverished families may not have telephone 
service or may move often. 

Out-of-school youth most often took advantage of job-skills training and paid or unpaid 
work experiences. Least accessed were child care or transportation assistance; nonetheless, nearly 
a third of participants reported receiving these services. 

Out-of-school youth were quite satisfied with the RESA I program. Their ratings for all 
three federally mandated satisfaction items exceeded the federal WIA performance objective. 

According to out-of-school respondents, program staff were knowledgeable and helpful. 
Respondents also received a lot of information through the program. On the other hand, 
respondents were somewhat less in agreement that they had experiences via the program that 
they would not have had otherwise. 

Out-of-school youth reported that the program was worthwhile and helpful, but that its 
services could be extended, more and more hands on training could be offered, and greater 
numbers of clients could be attracted to it. Youth reported currently needed assistance with 
seeking a job, obtaining job skills and a vehicle, and finding more information on post-secondary 
opportunities. 

Employers. Relatively few employers responded to the satisfaction survey, and the 
initial sample was somewhat limited as well. This, however, is likely because the program is in 
its early stages of development. One would expect higher levels of employer participation in the 
coming years. 

Employer satisfaction with the RESA I program exceeded the performance objective for 
this year of implementation. In general, employers were pleased with the program overall, 
thought that it had at least met their expectations, and that it began to approximate the ideal 
program for people in their circumstances. 
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Employers would recommend the program to other employers and plan to continue their 
participation in the coming year. On the other hand, some responding employers did not think 
that the process of providing work experiences to youth had been particularly easy. In addition, 
some respondents thought that the youth with whom they worked did not possess the needed 
skills for working in their organization. 

Recommendations 

• RESA I staff should continue to nurture their organizational culture of cooperation and 
collegiality. 

• Staff might want to contact other WIA programs to investigate strategies for addressing 
slow Job Service turnaround. Professional development experiences might be offered to 
staff around resolving this issue. 

• RESA I should continue to offer participants access to knowledgeable, helpful, and 
respectful staff. 

• Program staff should work to ensure that participants’ career and education goals are 
clearly articulated and that program services are deployed for their achievement. 

• Tutoring in specific content areas, assistance with job searches, and preparation for post- 
secondary experiences are services RESA I staff ought to target more fully to in-school 
youth. 

• RESA I staff might want to consider allowing some students to increase their work hours. 

• RESA I staff and local program staff should consider collaborating to offer enhanced job 
search and academic assistance to out-of-school youth. This might include more content- 
focused tutoring, on academic subjects or on job-seeking skills, as students evidence a 
need for it. 

• RESA I staff may want to investigate whether out-of-school youth’s concerns about the 
compensation structure and work hours are accurate. 

• To support employers as they provide work experiences for youth, RESA I and local staff 
should consider collaborating to offer training to youth in the areas employers report to be 
most important. These areas, at this writing, appear to concern social skills, good work 
habits, and more specific skill sets, such as computer, typing, calculation, or organization 
abilities. 

• As the program develops, RESA I staff ought to develop and implement structures and 
processes to ease employers’ participation. 
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RESA I staff and evaluators should explore strategies for increasing response rates to 
both mail and telephone surveys. Strategies might include beginning data collection at an 
earlier date in the year, continuing to build and update a comprehensive client database, 
and collecting alternative contact information for participants. 

Later evaluations of the RESA I program should include data from local service providers 
to provide staff a more complete account of program activities. 
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APPENDIX A: 



Satisfaction Surveys 



RES A I Workforce Investment Youth Services Program 
Spring 2002 Client Satisfaction Survey 
In-School Youth 



We are interested in continuously improving out services to you. Your responses to the questions on this 



1 . The following is a list of services you may have received through our program. Please check all 
of the services that you have received in the last year. 

Tutoring or study skill training 

Alternative high school 

Summer employment opportunities 

Paid or unpaid work experiences 

Job skills training 

Child care or transportation 

Mentoring 

Leadership training 

Follow-up services 

Guidance and counseling 

2. Using a scale of 1 to 10, with 1 meaning “very dissatisfied” and 10 meaning “very satisfied,” 
what is your overall level of satisfaction with the services provided by RESA I? 



services met those expectations? 1 now means “falls short of your expectations” and 10 means 
“exceeds expectations.” 



4. Now think of the ideal program for people in your circumstances. How well do you think the 

services you received compare with the ideal set of services? 1 now means “not very close to the 
ideal” and 10 means “very close to the ideal.” 



survey will give us information to better serve you. Please read each question carefully and then select 




Very 

Dissatisfied 

1 2 3 4 5 6 7 



Very 

Satisfied 



8 9 10 



3. Considering all of the expectations you may have had about the services, to what extent have the 



Falls Short of 
Expectations 



Exceeds 

Expectations 



1 2 3 4 5 



6 7 8 9 10 



Not Very Close 
to the Ideal 

1 2 3 4 5 



Very Close 
to the Ideal 



6 7 8 9 10 



PLEASE TURN SURVEY OVER TO COMPLETE 
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For the following questions, please use this scale: 

Strongly Strongly 

Disagree Disagree Agree Agree 

12 3 4 

5. The services I received helped me meet my education goals. 

6. The services I received helped me meet my career goals. 

7. I was involved in choosing the services that are right for me. 

8. The staff of this program were helpful. 

9. The staff of this program were knowledgeable. 

10. I felt that the staff respected me. 

11. I did not have to wait a long time to receive the services I needed. 

12. I received a lot of information through this program. 

13. Staff helped me find ways to participate in the program even if 
there were circumstances that made doing so difficult for me. 

14. I had experiences in this program that I would not have had 
otherwise. 

15. Staff were available to answer my questions. 

16. I feel more prepared for my future because of this program. 

17. I am glad I participated in this program. 

Please complete the following phrases. 

18. If I could change two things about the program, I would change . . . 



12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 



19. At this point in my experience with the program, I could use some help with . . . 



20. What school do you go to? 

21. Where do you receive Workforce Investment youth services? 



Thank you very much for your help! 
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RESA I Workforce Investment Youth Services Program 
Spring 2002 Client Satisfaction Phone Survey Protocol 
Out-of-School Youth 



Interviewee ID number: 

Date of interview: 

Time of interview: 

Hello. My name is with AEL, and I am conducting a survey for the RESA I Workforce 

Investment Youth Services program. I would like to speak to Mr./Ms. , please. 

[Continue if participant is answers the phone. If not, wait until participant comes to the phone, then greet 
with the above, and continue. If the participant is unavailable, ask when might be a good time to reach 
the participant. Record this information on the survey sheet, and return call at the appropriate time.] 

Hello, Mr./Ms. . My name is with AEL, and I am conducting a survey for the 

RESA I Workforce Investment Youth Services program. I would like to ask you some questions about 
your recent experience with the Youth Services program. Our purpose is to learn from you how to 
improve programs and services offered to youth in your county. The survey should take about 10 minutes. 
Is now a good time for you to participate? 

[If yes, continue. If no, ask when the participant would be able to participate. Record the answer on the 
survey sheet, and repeat the call at the appropriate time.] 

Thank you very much. First, let me assure you that your answers will be kept anonymous and 
confidential. That is, nothing you say will ever be associated with your name. Second, I want to let you 
know that there are no wrong answers. We really are interested in your opinions about this program. Do 
you have any questions? 

First, I am going to read a list of services you may have received. As I read them, indicate those that you 
remember receiving. 

[Interviewer should check each option the respondent selects.] 

1. 

. Tutoring or study skill training 

Alternative high school 

Summer employment opportunities 

Paid or unpaid work experiences 

Job skills training 

Child care or transportation 

Mentoring 

Leadership training 

Follow-up services 

Guidance and counseling 

Now I am going to ask you to rate your answers to the following questions. [Interviewer should circle the 
rating the respondent chooses.] 
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2 . 



Using a scale of 1 to 10, with 1 meaning “very dissatisfied” and 10 meaning “very satisfied,” what 
is your overall level of satisfaction with the services provided by RESA I staff? 

123456789 10 Don’t Know Refused 

3. Considering all of the expectations you may have had about the services, to what extent have the 
services met those expectations? 1 now means “falls short of your expectations” and 10 means 
^exceeds expectations.” 

123456789 10 Don’t Know Refused 

4. Now think of the ideal program for people in your circumstances. How well do you think the 
services you received compare with the ideal set of services? 1 now means “not very close to the 
ideal” and 10 means “very close to the ideal.” 

123456789 10 Don’t Know Refused 



For the next set of questions, I am going to read you some statements. Please tell me how much you agree 
with each statement, using a scale of 1 to 4, with 1 meaning “strongly disagree,” 2 meaning “disagree,” 3 
meaning “agree,” and 4 meaning “strongly agree.” 

SD D A SA 



5. 


The services I received helped me meet my education goals. 


1 


2 


3 


4 


DK 


Ref 


6. 


The services I received helped me meet my career goals. 


1 


2 


3 


4 


DK 


Ref 


7. 


I was involved in choosing the services that are right for me. 


1 


2 


3 


4 


DK 


Ref 


8. 


The staff of this program were helpful. 


1 


2 


3 


4 


DK 


Ref 


9. 


The staff of this program were knowledgeable. 


1 


2 


3 


4 


DK 


Ref 


10. 


I felt that the staff respected me. 


1 


2 


3 


4 


DK 


Ref 


11. 


I did not have to wait a long time to receive the services I needed. 


1 


2 


3 


4 


DK 


Ref 


12. 


I received a lot of information through this program. 


1 


2 


3 


4 


DK 


Ref 


13. 


Staff helped me find ways to participate in the program even if 
there were circumstances that made doing so difficult for me. 


1 


2 


3 


4 


DK 


Ref 


14. 


I had experiences in this program that I would not have had 
otherwise. 


1 


2 


3 


4 


DK 


Ref 


15. 


Staff were available to answer my questions. 


1 


2 


3 


4 


DK 


Ref 


16. 


I feel more prepared for my future because of this program. 


1 


2 


3 


4 


DK 


Ref 


17. 


I am glad I participated in this program. 


1 


2 


3 


4 


DK 


Ref 



Now I am going to ask you to complete several phrases. 

18. If I could change two things about the program, I would change . . . 



19. At this point in my experience with the program, I could use some help with . . . 



O , 
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are all the questions I have today. Thank you very much for talking with me. 



RESA I Workforce Investment Youth Services Program 
Spring 2002 Employer Satisfaction Phone Survey Protocol 



Hello. My name is with AEL, and I am conducting a survey for the RESA I Workforce 

Investment Youth Services program. I would like to speak to Mr./Ms. , please. 

[Continue if participant is answers the phone. If not, wait until participant comes to the phone, then greet 
with the above, and continue. If the participant is unavailable, ask when might be a good time to reach 
the participant. Record this information on the survey sheet, and return call at the appropriate time.] 

Hello, Mr./Ms. . My name is with AEL, and I am conducting a survey for the 

RESA I Workforce Investment Youth Services program. I would like to ask you some questions about 
your recent experience with the Youth Services program. Our purpose is to learn from you how to 
improve programs and services offered to employers and youth in your county. The survey should take 
about 10 minutes. Is now a good time for you to participate? 

[If yes, continue. If no, ask when the participant would be able to participate. Record the answer on the 
survey sheet, and repeat the call at the appropriate time.] 

Thank you very much. First, let me assure you that your answers will be kept anonymous and 
confidential. That is, nothing you say will ever be associated with your name. Second, I want to let you 
know that there are no wrong answers. We really are interested in your opinions about this program. Do 
you have any questions before we begin? 

1. Which of the following did you provide to youth this last year in collaboration with the RESA I 
Youth Program? (Choose all that apply.) 

Unpaid work experience (e.g., job shadowing) 

Paid work experience (e.g., work-based learning) 

2. Using a scale of 1 to 10, with 1 meaning “very dissatisfied” and 10 meaning “very satisfied,” what 
is your overall level of satisfaction with the services provided by RESA I staff? 

123456789 10 Don’t Know Refused 

3. Considering all of the expectations you may have had about the services, to what extent have the 
services from RESA I staff met those expectations? 1 now means “falls short of your 
expectations” and 10 means “exceeds expectations.” 

123456789 10 Don’t Know Refused 

4. Now think of the ideal program for people in your circumstances. How well do you think the 
services you received compare with the ideal set of services? 1 now means “not very close to the 
ideal” and 10 means “very close to the ideal.” 

123456789 10 Don’t Know Refused 
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For the next set of questions, I am going to read you some statements. Please tell me how much you agree 
with each statement, using a scale of 1 to 4, with 1 meaning “strongly disagree,” 2 meaning “disagree,” 3 
meaning “agree,” and 4 meaning “strongly agree.” 



5. 


The RESA I Youth Services staff communicated the purpose of 

the program to me clearly. 1 2 


3 


4 


DK 


Ref 


6. 


The RESA I Youth Services staff were helpful. 1 2 


3 


4 


DK 


Ref 


7. 


The RESA I Youth Services staff understood my needs as an 
employer. 1 2 


3 


4 


DK 


Ref 


8. 


When I had questions or concerns about the program, RESA I 

staff responded quickly to them. 1 2 


3 


4 


DK 


Ref 


9. 


I received adequate support from RESA I throughout the process 
of offering work experiences to youth. 1 2 


3 


4 


DK 


Ref 


10. 


The process of providing work experiences to youth has been easy. 1 2 


3 


4 


DK 


Ref 


11. 


The youth I worked with had the skills I thought they needed for 
employment in my organization. 1 2 


3 


4 


DK 


Ref 


12. 


The youth I worked with were well-behaved. 1 2 


3 


4 


DK 


Ref 


13. 


I think experiences like the one(s) I provided will help youth 

in my area become better prepared for employment. 1 2 


3 


4 


DK 


Ref 


14. 


I would recommend this program to other employers. 1 2 


3 


4 


DK 


Ref 


15, 


I plan to continue participating in this program. 1 2 


3 


4 


DK 


Ref 



Now I am going to ask you to complete several sentences. 

16. Youth need to come to work experiences like the one(s) I provided with the following job skills: 



17. It would be easier for me to participate in this program if . . . 



Thank you for answering our questions. We appreciate it very much. 
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Checklist for Applying the Standards 



To interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear in Joint 
Committee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, CA, Sage. 

The Standards were consulted and used as indicated in the table below (check as appropriate): 



Descriptor 

U 1 Stakeholder Identification 


The Standard was 
addressed 

v 


The Standard was 
partially addressed 


The Standard was 
not addressed 


The Standard was 
not applicable 


U2 


Evaluator Credibility 


X 








U3 


Information Scope and Selection 


X 








U4 


Values Identification 


X 








U5 


Report Clarity 


X 








U6 


Report Timeliness and Dissemination 


X 








U7 


Evaluation Impact 


X 








FI 


Practical Procedures 


X 








F2 


Political Viability 








X 


F3 


Cost Effectiveness 


X 








PI 


Service Orientation 


X 








P2 


Formal Agreements 


X 








P3 


Rights of Human Subjects 


X 








P4 


Human Interactions 


X 








P5 


Complete and Fair Assessment 


X 








P6 


Disclosure of Findings 


X 








P7 


Conflict of Interest 


X 








P8 


Fiscal Responsibility 


X 








A1 


Program Documentation 


X 








A2 


Context Analysis 


X 








A3 


Described Purposes and Procedures 


X 








A4 


Defensible Information Sources 


X 








A5 


Valid Information 


X 








A6 


Reliable Information 


X 








A7 


Systematic Information 


X 








A8 


Analysis of Quantitative Information 


X 








A9 


Analysis of Qualitative Information 


X 








A10 


Justified Conclusions 


X 








All 


Impartial Reporting 


X 








A12 


Metaevaluation 






X 





The Program Evaluation Standards (1994, Sage) guided the development of this (check one): 

request for evaluation plan/design/proposal 

evaluation plan/design/proposal 

evaluation contract 

X evaluation report 

other: 



Name Caitlin Howley 



Date 



Caitlin Howlev 






u 



Position or Title R&E Specialis t 



(signature) 



Agency . 
Address = 



AEL 



PO Box 1348, Charleston, WV 25325-1348 



Relation to Document Author 



(c.g., author of document, evaluation team leader, external auditor, internal auditor) 



ERIC 
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